recordExtractor

What to extract from crawled pages and how to format it as Algolia records.

The recordExtractor action is a function that takes an object as an input parameter and returns a list of records.

JavaScript

recordExtractor: ({ url, $, contentLength, fileType})  => {
return [
    {
    url: url.href,
    text: $('p').html()
    ... /* Anything you want */
    }
];
// return []; skips the page
}

Parameters

Specify one or more response parameters in your recordExtractor to determine what information is returned.

$ · object

A Cheerio instance with the HTML of the crawled page. For more information, see Extracting data with Cheerio.

contentLength · number

The size of the crawled page in bytes.

dataSources · object

The external data sources of the current URL. Each key of this object corresponds to an externalData object. For example:

JavaScript

{
  dataSources: {
    dataSourceId1: { data1: 'val1', data2: 'val2' },
    dataSourceId2: { data1: 'val1', data2: 'val2' },
  }
}

filetype · string

The file type of the crawled page or document.

helpers · function

Helpers are functions that help extract content and generate records. This can help simplify your record extractor.

url · object

A Location object that contains the URL.

Returns

The record extractor returns an array of records with attributes or an empty array. If it returns an empty array, the page is skipped (isn’t crawled).